Liberating Structures in Strategic Planning: Methods to Foster Engagement.

This paper describes the innovative approach of using liberating structures to the development of the AUR 2023 strategic plan, and lessons learned in their application. The 2023 strategic plan built on the results and approach of the prior 2015 plan. Similar to the 2015 strategic plan, traditional tools such as a SWOT analysis and strategic retreat were used. In addition, the 2023 process included tools called liberating structures and was iteratively co-produced through a series of virtual meetings over 18 months. Advantages of liberating structures included increased creativity and speed in moving through meeting tasks, increased number of meaningful contributions from AUR members and increased engagement from participants during discussions and meetings. The 2023 AUR strategic plan is provided along with examples of completed goals and those under early implementation. Lessons learned from using these tools for strategic planning can be applied to other society and group meetings. Moving forward, the 2023 strategic plan will be a living document, which will be reviewed at each Board of Directors meeting and periodically adapted.

Quality Improvement Report: Design and Implementation of a Radiology E-Consult Service.

Electronic consultations (e-consults) mediated through an electronic health record system or web-based platform allow synchronous or asynchronous physician-to-physician communication. E-consults have been explored in various clinical specialties, but relatively few instances in the literature describe e-consults to connect health care providers directly with radiologists.The authors outline how a radiology department can implement an e-consult service and review the development of such a service in a large academic health system. They describe the logistics, workflow, turnaround time expectations, stakeholder management, and pilot implementation and highlight challenges and lessons learned.

Impact of Artificial Intelligence-Assisted Indication Selection on Appropriateness Order Scoring for Imaging Clinical Decision Support.

PURPOSE: The aim of this study was to assess appropriateness scoring and structured order entry after the implementation of an artificial intelligence (AI) tool for analysis of free-text indications. METHODS: Advanced outpatient imaging orders in a multicenter health care system were recorded 7 months before (March 1, 2020, to September 21, 2020) and after (October 20, 2020, to May 13, 2021) the implementation of an AI tool targeting free-text indications. Clinical decision support score (not appropriate, may be appropriate, appropriate, or unscored) and indication type (structured, free-text, both, or none) were assessed. The χ2 and multivariate logistic regression adjusting for covariables with bootstrapping were used. RESULTS: In total, 115,079 orders before and 150,950 orders after AI tool deployment were analyzed. The mean patient age was 59.3 ± 15.5 years, and 146,035 (54.9%) were women; 49.9% of orders were for CT, 38.8% for MR, 5.9% for nuclear medicine, and 5.4% for PET. After deployment, scored orders increased to 52% from 30% (P < .001). Orders with structured indications increased to 67.3% from 34.6% (P < .001). On multivariate analysis, orders were more likely to be scored after tool deployment (odds ratio [OR], 2.7, 95% CI, 2.63-2.78; P < .001). Compared with physicians, orders placed by nonphysician providers were less likely to be scored (OR, 0.80; 95% CI, 0.78-0.83; P < .001). MR (OR, 0.84; 95% CI, 0.82-0.87) and PET (OR, 0.12; 95% CI, 0.10-0.13) were less likely to be scored than CT (; P < .001). After AI tool deployment, 72,083 orders (47.8%) remained unscored, 45,186 (62.7%) with free-text-only indications. CONCLUSIONS: Embedding AI assistance within imaging clinical decision support was associated with increased structured indication orders and independently predicted a higher likelihood of scored orders. However, 48% of orders remained unscored, driven by both provider behavior and infrastructure-related barriers.

Building Effective Teams in the Real World From Traps to Triumph.

Quality patient care and advancements in medical education, investigation, and innovation require effective teamwork. High-functioning teams navigate stressful environments, learning openly from failures and leveraging successes to fuel future initiatives. The authors review foundational concepts for implementing and sustaining successful teams, including emotional intelligence, trust, inclusivity, clear communication, and accountability. Focus is given to real-world examples and actionable, practical solutions.

Diagnostic Performance of the Ovarian-Adnexal Reporting and Data System (O-RADS) Ultrasound Risk Score in Women in the United States.

Importance: The American College of Radiology (ACR) Ovarian-Adnexal Reporting and Data System (O-RADS) ultrasound (US) risk scoring system has been studied in a selected population of women referred for suspected or known adnexal lesions. This population has a higher frequency of malignant neoplasms than women presenting to radiology departments for pelvic ultrasonography for a variety of indications, potentially impacting the diagnostic performance of the risk scoring system. Objective: To evaluate the risk of malignant neoplasm and diagnostic performance of O-RADS US risk scoring system in a multi-institutional, nonselected cohort. Design, Setting, and Participants: This multi-institutional cohort study included a population of nonselected women in the United States who presented to radiology departments for routine pelvic ultrasonography between 2011 and 2014, with pathology confirmation imaging follow up or 2 years of clinical follow up. Exposure: Analysis of 1014 adnexal lesions using the O-RADS US risk stratification system. Main Outcomes and Measures: Frequency of ovarian cancer and diagnostic performance of the O-RADS US risk stratification system. Results: This study included 913 women with 1014 adnexal lesions. The mean (SD) age of the patients was 42.4 (13.9 years), and 674 of 913 (73.8%) were premenopausal. The overall frequency of malignant neoplasm was 8.4% (85 of 1014 adnexal lesions). The frequency of malignant neoplasm for O-RADS US 2 was 0.5% (3 of 657 lesions; <1% expected); O-RADS US 3, 4.5% (5 of 112 lesions; <10% expected); O-RADS US 4, 11.6% (18 of 155; 10%-50% expected); and O-RADS 5, 65.6% (59 of 90 lesions; >50% expected). O-RADS US 4 was the optimum cutoff for diagnosing cancer with sensitivity of 90.6% (95% CI, 82.3%-95.9%), specificity of 81.9% (95% CI, 79.3%-84.3%), positive predictive value of 31.4% (95% CI, 25.7%-37.7%) and negative predictive value of 99.0% (95% CI, 98.0%-99.6%). Conclusions and Relevance: In this cohort study of a nonselected patient population, the O-RADS US risk stratification system performed within the expected range as published by the ACR O-RADS US committee. The frequency of malignant neoplasm was at the lower end of the published range, partially because of the lower prevalence of cancer in a nonselected population. However, a high negative predictive value was maintained, and when a lesion can be classified as an O-RADS US 2, the risk of cancer is low, which is reassuring for both clinician and patient.

Development and Implementation of a Multisite Registry Using Structured Templates for Actionable Findings in the Kidney.

PURPOSE: The aim of this study was to scale structured report templates categorizing actionable renal findings across health systems and create a centralized registry of patient and report data. METHODS: In January 2017, three academic radiology departments agreed to prospectively include identical structured templates categorizing the malignant likelihood of renal findings in ≥90% of all adult ultrasound, MRI, and CT reports, a new approach for two sites. Between November 20, 2017, and September 30, 2019, deidentified HL7 report data were transmitted to a centralized ACR registry. An automated algorithm extracted categories. Radiologists were requested to addend reports with missing or incomplete templates after the first month. Separately, each site submitted patient sociodemographic and clinical data 12 months before and at least 3 months after enrollment. RESULTS: A total of 164,982 eligible radiology reports were transmitted to the registry; 4,159 (2.5%) were excluded because of missing categories or radiologist names. The final cohort included 160,823 examinations on 102,619 unique patients. Mean template use before and after addendum requests was 99.3% and 99.9% at SITE1, 86.5% and 94.6% at SITE2, and 91.4% and 96.0% at SITE3. Matching patient sociodemographic and clinical data were obtained on 96.9% of reports from SITE1, 94.2% from SITE2, and 96.0% from SITE3. Regulatory, cultural, and technology barriers to the creation of a multisite registry were identified. CONCLUSIONS: Barriers to the adoption of unified structured report templates for actionable kidney findings can be addressed. Deidentified report and patient data can be securely transmitted to an external registry. These data can facilitate the collection of diverse evidence-based population imaging outcomes.

Ovarian Cancer Detection in Average-Risk Women: Classic- versus Nonclassic-appearing Adnexal Lesions at US.

Background Several US risk stratification schemas for assessing adnexal lesions exist. These multiple-subcategory systems may be more multifaceted than necessary for isolated adnexal lesions in average-risk women. Purpose To explore whether a US-based classification scheme of classic versus nonclassic appearance can be used to help appropriately triage women at average risk of ovarian cancer without compromising diagnostic performance. Materials and Methods This retrospective multicenter study included isolated ovarian lesions identified at pelvic US performed between January 2011 and June 2014, reviewed between September 2019 and September 2020. Lesions were considered isolated in the absence of ascites or peritoneal implants. Lesions were classified as classic or nonclassic based on sonographic appearance. Classic lesions included simple cysts, hemorrhagic cysts, endometriomas, and dermoids. Otherwise, lesions were considered nonclassic. Outcomes based on histopathologic results or clinical or imaging follow-up were recorded. Diagnostic performance and frequency of malignancy were calculated. Frequency of malignancy between age groups was compared using the χ2 test, and Poisson regression was used to explore relationships between imaging features and malignancy. Results A total of 970 isolated lesions in 878 women (mean age, 42 years ± 14 [SD]) were included. The malignancy rate for classic lesions was less than 1%. Of 970 lesions, 53 (6%) were malignant. The malignancy rate for nonclassic lesions was 32% (33 of 103) when blood flow was present and 8% (16 of 194) without blood flow (P < .001). For women older than 60 years, the malignancy rate was 50% (10 of 20 lesions) when blood flow was present and 13% (five of 38) without blood flow (P = .004). The sensitivity, specificity, positive predictive value, and negative predictive value of the classic-versus-nonclassic schema was 93% (49 of 53 lesions), 73% (669 of 917 lesions), 17% (49 of 297 lesions), and 99% (669 of 673 lesions), respectively, for detection of malignancy. Conclusion Using a US classification schema of classic- or nonclassic-appearing adnexal lesions resulted in high sensitivity and specificity in the diagnosis of malignancy in ovarian cancer. The highest risk of cancer was in isolated nonclassic lesions with blood flow in women older than 60 years. © RSNA, 2022 See also the editorial by Baumgarten in this issue.

Patient Understanding of Abnormal Imaging Findings Under Pennsylvania Act 112: A Call to Revise Mandated Notification Message Language.

PURPOSE: The aim of this study was to evaluate the effect of Pennsylvania Act 112 notification reading level and presentation on patient understanding and anxiety. METHODS: Four notifications were developed by alternating 12th grade and 6th grade reading level Act 112 language with letters or infographics. Using Amazon Mechanical Turk, 909 US adult volunteers were randomly assigned to one notification followed by a survey. Participants who answered all 12 survey questions on understanding, anxiety, and sociodemographics were paid $0.10. Chi-square analysis and multivariate regression were used to determine the impact of notification type and sociodemographic data on understanding of communicated information and anxiety. RESULTS: Sixty percent of participants (489 of 821) correctly understood all three questions directly answered within notifications regarding Act 112 subject, next steps, and process for obtaining reports. Approximately half of respondents understood that notifications indirectly conveyed "definitely" or "possibly" abnormal test results (344 of 821 [42%] and 99 of 821 [12%], respectively). Compared with the 12th grade letter, correct understanding of all directly communicated information was lower with the 12th grade infographic after adjustment (odds ratio, 0.61; 95% confidence interval, 0.39-0.95; P = .028) and equivalent with the 6th grade infographic and letter (P = .744 and P = .316). Correct indirect understanding of abnormal test results was not associated with notification type after adjustment but was associated with higher anxiety (odds ratio, 2.86; 95% confidence interval, 0.57-1.35; P < .001). CONCLUSIONS: Layperson understanding of information directly and indirectly communicated in Pennsylvania Act 112 is suboptimal, regardless of reading level or presentation. New Act 112 language is needed to improve patient understanding, which would ideally be coproduced with Pennsylvania patients, policymakers, and other relevant stakeholders.

Lessons From the Free-Text Epidemic: Opportunities to Optimize Deployment of Imaging Clinical Decision Support.

OBJECTIVE: The Protecting Access to Medicare Act of 2014 requires clinicians to consult Appropriate Use Criteria (AUC) when ordering advanced imaging procedures. Free-text order indications are available when there is no applicable structured indication but are unscored by the AUC. We determined the proportion of free-text indications among all advanced imaging orders and the proportion of free-text indications that could be mapped to a single structured indication. METHODS: All outpatient advanced diagnostic imaging orders placed in a large multisite health system were recorded after initial AUC deployment (November 20, 2017, to December 19, 2017). Clinicians were prompted upon order entry to select a structured indication or enter a free-text indication. We manually reviewed the two imaging examinations with the highest rate of free-text indications: enhanced CT abdomen/pelvis and unenhanced CT head. Regression analysis examined differences in patient-, imaging-, context-, and provider-level characteristics between scored and unscored examinations. RESULTS: Among all 39,533 orders for advanced imaging procedures, 59% (23,267 of 39,533) were unscored by the system. The regression model c-statistic (0.50-0.55) demonstrated poor model fit to evaluate for differences between scored and unscored examinations. Free-text indications were found in 71% (16,440 of 23,267) of unscored examinations and 42% (16,440 of 39,533) of all examinations. Manual review of all 1,693 CT abdomen/pelvis and 1,527 CT head examinations with free-text indications revealed that 3,132 free-text indications (97%) could be mapped to a single existing structured indication. DISCUSSION: Of all initially placed outpatient advanced imaging procedure orders, 42% included free-text indications and 97% of manually reviewed free-text indications could be mapped to a single structured indication.

Early Impact of Pennsylvania Act 112 on Follow-up of Abnormal Imaging Findings.

OBJECTIVE: Pennsylvania Act 112 requires diagnostic imaging facilities to directly notify outpatients about significant imaging abnormalities that require follow-up care within 3 months. The effects of Act 112 on patient care are unclear. We sought to characterize follow-up discussions and care received by outpatients with significant imaging abnormalities as defined by Act 112. METHODS: We evaluated findings flagged for patient notification under Act 112 at our institution over a 1-month period. We analyzed findings for radiologic reporting, follow-up discussions between patients and ordering providers, and follow-up medical care provided. RESULTS: Follow-up discussions were documented for 87% of findings (n = 205 of 235) and occurred on average 6.0 days after imaging examinations were performed. Follow-up discussions directly attributable to the Act 112 letter occurred in 0.4% of findings. Follow-up care was provided for 74% of findings on average 31.3 days after imaging examinations were performed. Provider-initiated follow-up discussions occurred earlier and were associated with shorter time to follow-up care when compared with patient-initiated discussions. Direct contact of ordering provider by interpreting radiologist was a significant predictor of occurrence of follow-up discussions and length of time to follow-up care. DISCUSSION: Act 112 had a small impact at our institution on improving completed follow-up for abnormal imaging findings. Our results also imply that health systems should encourage timeliness of patient-provider discussions of abnormal imaging findings and facilitate direct radiologist communication with ordering providers. Future studies should evaluate the impact of Act 112 in different practice settings to understand its broader impact on follow-up care.

Use of an Online Crowdsourcing Platform to Assess Patient Comprehension of Radiology Reports and Colloquialisms.

OBJECTIVE. The purpose of this study was to use an online crowdsourcing platform to assess patient comprehension of five radiology reporting templates and radiology colloquialisms. MATERIALS AND METHODS. In this cross-sectional study, participants were surveyed as patient surrogates using a crowdsourcing platform. Two tasks were completed within two 48-hour time periods. For the first crowdsourcing task, each participant was randomly assigned a set of radiology reports in a constructed reporting template and subsequently tested for comprehension. For the second crowdsourcing task, each participant was randomly assigned a radiology colloquialism and asked to indicate whether the phrase indicated a normal, abnormal, or ambivalent finding. RESULTS. A total of 203 participants enrolled for the first task and 1166 for the second within 48 hours of task publication. The payment totaled $31.96. Of 812 radiology reports read, 384 (47%) were correctly interpreted by the patient surrogates. Patient surrogates had higher rates of comprehension of reports written in the patient summary (57%, p < 0.001) and traditional unstructured in combination with patient summary (51%, p = 0.004) formats than in the traditional unstructured format (40%). Most of the patient surrogates (114/203 [56%]) expressed a preference for receiving a full radiology report via an electronic patient portal. Several radiology colloquialisms with modifiers such as "low," "underdistended," and "decompressed" had low rates of comprehension. CONCLUSION. Use of the crowdsourcing platform is an expeditious, cost-effective, and customizable tool for surveying laypeople in sentiment- or task-based research. Patient summaries can help increase patient comprehension of radiology reports. Radiology colloquialisms are likely to be misunderstood by patients.

Automated Detection of Radiology Reports that Require Follow-up Imaging Using Natural Language Processing Feature Engineering and Machine Learning Classification.

While radiologists regularly issue follow-up recommendations, our preliminary research has shown that anywhere from 35 to 50% of patients who receive follow-up recommendations for findings of possible cancer on abdominopelvic imaging do not return for follow-up. As such, they remain at risk for adverse outcomes related to missed or delayed cancer diagnosis. In this study, we develop an algorithm to automatically detect free text radiology reports that have a follow-up recommendation using natural language processing (NLP) techniques and machine learning models. The data set used in this study consists of 6000 free text reports from the author's institution. NLP techniques are used to engineer 1500 features, which include the most informative unigrams, bigrams, and trigrams in the training corpus after performing tokenization and Porter stemming. On this data set, we train naive Bayes, decision tree, and maximum entropy models. The decision tree model, with an F1 score of 0.458 and accuracy of 0.862, outperforms both the naive Bayes (F1 score of 0.381) and maximum entropy (F1 score of 0.387) models. The models were analyzed to determine predictive features, with term frequency of n-grams such as "renal neoplasm" and "evalu with enhanc" being most predictive of a follow-up recommendation. Key to maximizing performance was feature engineering that extracts predictive information and appropriate selection of machine learning algorithms based on the feature set.

Toward Complete Structured Information Extraction from Radiology Reports Using Machine Learning.

Unstructured and semi-structured radiology reports represent an underutilized trove of information for machine learning (ML)-based clinical informatics applications, including abnormality tracking systems, research cohort identification, point-of-care summarization, semi-automated report writing, and as a source of weak data labels for training image processing systems. Clinical ML systems must be interpretable to ensure user trust. To create interpretable models applicable to all of these tasks, we can build general-purpose systems which extract all relevant human-level assertions or "facts" documented in reports; identifying these facts is an information extraction (IE) task. Previous IE work in radiology has focused on a limited set of information, and extracts isolated entities (i.e., single words such as "lesion" or "cyst") rather than complete facts, which require the linking of multiple entities and modifiers. Here, we develop a prototype system to extract all useful information in abdominopelvic radiology reports (findings, recommendations, clinical history, procedures, imaging indications and limitations, etc.), in the form of complete, contextualized facts. We construct an information schema to capture the bulk of information in reports, develop real-time ML models to extract this information, and demonstrate the feasibility and performance of the system.

The key radiologic and cytomorphologic features of oncocytic and oncocytoid lesions of the salivary gland.

Oncocytic and oncocytoid lesions represent a distinct subset of salivary gland lesions. True oncocytic lesions of the salivary gland are entirely composed of oncocytes. These are characterized by the presence of abundant eosinophilic granules due to the presence of abundant cytoplasmic mitochondria. Oncocytic lesions of the salivary gland include oncocytosis, oncocytoma, and oncocytic carcinoma. In addition to the true oncocytic lesion, there exists another group of salivary gland lesions, which demonstrate cells with abundant and occasionally granular cytoplasm. These are often termed as "oncocytoid" lesions. The recently proposed Milan System for reporting salivary gland cytology clearly states that fine-needle aspiration specimens representing oncocytic/oncocytoid lesions of salivary gland cannot effectively distinguish between a nonneoplastic lesion, benign and malignant neoplasms. Therefore, most lesions lacking classic cytomorphologic features will be classified under the umbrella diagnostic term of "Salivary Gland Neoplasm of Uncertain Malignant Potential" (SUMP). In this review, we discuss and illustrate key clinicopathologic and radiologic features that can help the practicing cytopathologist narrow down the differential and provide the best management based diagnosis.

Location, Location, Location: The Association Between Imaging Setting and Follow-Up of Findings of Indeterminate Malignant Potential.

PURPOSE: To evaluate the relationship between patient location at time of imaging and completion of relevant imaging follow-up for findings with indeterminate malignant potential. METHODS: We used a mandatory hospital-wide standardized assessment categorization system to analyze all ultrasound, CT, and MRI examinations performed over a 7-month period. Multivariate logistic regression, adjusted for imaging modality, characteristics of patients, ordering clinicians, and interpreting radiologists, was used to evaluate the relationship between patient location (outpatient, inpatient, or emergency department) at the time of index examination and completion of relevant outpatient imaging follow-up. RESULTS: Relevant follow-up occurred in 49% of index examinations, with a greater percentage among those performed in the outpatient setting compared with those performed in the inpatient or emergency department settings (62% versus 18% versus 17%, respectively). Compared with examinations obtained in the outpatient setting, examinations performed in the emergency department (adjusted odds ratio [aOR] 0.07; 95% confidence interval [CI], 0.03-0.19) and inpatient (aOR 0.14; 95% CI, 0.09-0.23) settings were less likely to be followed up. Black patients and those residing in lower-income neighborhoods were also less likely to receive relevant follow-up. Few lesions progressed to more suspicious lesions (4.6%). CONCLUSIONS: Patient location at time of imaging is associated with the likelihood of completing relevant follow-up imaging for lesions with indeterminate malignant potential. Future work should evaluate health system-level care processes related to care setting, as well as their effects on appropriate follow-up imaging. Doing so would support efforts to improve appropriate follow-up imaging and reduce health care disparities.

Patient Factor Disparities in Imaging Follow-Up Rates After Incidental Abdominal Findings.

OBJECTIVE: The effect of demographics and societal determinants on imaging follow-up rates is not clear. The purpose of this study was to compare characteristics of patients with imaging findings representing possible cancer who undergo follow-up imaging versus those who do not to better understand factors that contribute to follow-up completion. MATERIALS AND METHODS: The records of 1588 patients with indeterminate abdominal imaging findings consecutively registered between July 1, 2013, and March 20, 2014, were reviewed. Several patient characteristics, including distance between patients' home zip codes and the flagship hospital of the health system were compared between the groups who did and did not undergo follow-up imaging. Subgroup analyses based on the location of the index examination were also performed. RESULTS: Among the 1513 (36.62%) included patients, 554 did not undergo follow-up abdominal imaging within 1 year of the index examination. The same was true of 270 of 938 (28.78%) outpatients and 168 of 279 (60.21%) emergency department patients. Eighty-nine of 959 (9.28%) patients who underwent follow-up imaging were younger than 40 years, compared with 76 of 554 (13.72%) patients who did not undergo follow-up imaging (p = 0.005). Fifty-four of 959 (5.63%) patients who underwent follow-up imaging were older than 80 years, compared with 70 of 554 (12.64%) patients who did not undergo follow-up imaging (p < 0.001). More white patients (587 of 959 vs 301 of 554, p = 0.007) and fewer black patients (204 of 554 versus 270 of 959, p < 0.001) were found in the follow-up imaging group. Greater distance from the flagship hospital correlated with less follow-up in the outpatient subgroup only (p = 0.03). CONCLUSION: Emergency department patients and patients at the extremes of age are less likely to complete follow-up imaging. Insurance status and race and ethnicity may affect follow-up completion rates. The relationship between distance to hospital and follow-up completion requires further investigation.